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A GRAPHICAL USER INTERFACE FOR DIGITAL MEDIA AND NETWORK 
PORTALS USING DETAIL-IN-CONTEXT LENSES 

The invention relates to the field of computer graphics processing, and more specifically to a 
graphical user interface using detail-in-context lenses for direct interaction with digital media 
5 and for interfacing with network portals. 

BACKGROUND OF THE INVENTION 

Display screens are the primary visual display interface to a computer. One problem with these 
visual display screens is that they are limited in size, thus presenting a challenge to user interface 
design, particularly when largo- amounts of information is to be displayed. This problem is 
1 0 normally referred to as the "screen real estate problem". 

Well-known solutions to this problem include panning, zooming, scrolling or combinations 
thereof. While these solutions are suitable for a large number of visual display applications, 
these solutions become less effective where the visual information is spatially related, such as 
maps, newspapers and such like. In this type of information display, panning, zooming and/or 
IS scrolling is not as effective as much of the context of the panned, zoomed or scrolled display is 
hidden. 

A recent solution to this problem is the application of "detail-in-context" presentation 
techniques. Detail-in-context is the magnification of a particular region of interest (the "focal 
region" or "detail") in a data presentation while preserving visibility of the surrounding 
20 information (the "context"). This technique has applicability to the display of large surface area 
media, such as maps, on limited size computer screens including laptop computers, personal 
digital assistants ("PDAs"), and cell phones. 

In the detail-in-context discourse, differentiation is often made between the terms 
"representation" and "presentation". A representation is a formal system, or mapping, for 
25 specifying raw information or data that is stored in a computer or data processing system. For 
example, a digital map of a city is a representation of raw data including street names and the 
relative geographic location of streets and utilities. Such a representation may be displayed 
visually on a computer screen or printed on paper. On the other hand, a presentation is a spatial 
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organization of a given representation that is appropriate for the task at hand. Thus, a 
presentation of a representation organizes such things as the point of view and the relative 
emphasis of different parts or regions of the representation. For example, a digital map of a city 
maybe presented with a region magnified to reveal street names. 

5 In general, a detail-in-context presentation may be considered as a distorted view (or distortion) 
of a portion of the original representation where the distortion is the result of the application of a 
"lens" like distortion function to the original representation. A detailed review of various detail- 
in-context presentation techniques such as Elastic Presentation Space may be found in a 
publication by Marianne S. T. Carpendale, entitled "A Framework for Elastic Presentation 
10 Space" (Carpendale, Marianne S. T., A Framework for Elastic Presentation Space (Burnaby, 
British Columbia: Simon Fraser University, 1999)), and incorporated herein by reference. 

In general, detail-in-context data presentations are characterized by magnification of areas of an 
image where detail is desired, in combination with compression of a restricted range of areas of 
the remaining information (i.e. the context), the result typically giving the appearance of a lens 

1 5 having been applied to the display surface. Using the techniques described by Carpendale, points 
in a representation are displaced in three dimensions and a perspective projection is used to 
display the points on a two-dimensional presentation display. In detail-in-context presentation 
systems, when a lens is applied to a two-dimensional continuous surface representation, for 
example, the resulting presentation appears to be three-dimensional. In other words, the lens 

20 transformation appears to have stretched the continuous surface in a third dimension. 

One shortcoming of present graphical user interfaces for interaction with digital media and for 
interfacing with network portals is that they do not allow for effective user control of detail-in- 
context presentations. 

A need therefore exists for a graphical user interface for digital media and for interfacing with 
25 network portals that allows for effective user control of detail-in-context presentations. 
Consequently, it is an object of the present invention to obviate or mitigate at least some of the 
above mentioned disadvantages. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

Embodiments of the invention may best be understood by referring to the following description 
and accompanying drawings. In the description and drawings, line numerals refer to like 
structures or processes. In the drawings: 

S FIG. 1 is a graphical representation of the geometry for constructing a three-dimensional (3D) 
perspective viewing frustum, relative to an x, y, z coordinate system, in accordance with known 
elastic presentation space graphics technology; 

FIG. 2 is a graphical representation of the geometry of a presentation in accordance with known 
elastic presentation space graphics technology; 

1 0 FIG. 3 is a block diagram illustrating an exemplary data processing system for implementing an 
embodiment of the invention; 

FIG. 4 is a screen capture illustrating a detail-in-context lens applied to a celebrity's wrist watch 
in the context of a television program in accordance with an embodiment of the invention; 

FIG. 5 is a screen capture illustrating a lens applied to advertisements in a video game in 
1 5 accordance with an embodiment of the invention; and, 

FIG. 6 is a screen capture illustrating the application of lenses in multi-player video games in 
accordance with an embodiment of the invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

In the following description, numerous specific details are set forth to provide a thorough 
20 understanding of the invention. However, it is understood that the invention may be practiced 
without these specific details. In other instances, well-known software, circuits, structures and 
techniques have not been described or shown in detail in order not to obscure the invention. In 
the drawings, like numerals refer to like structures or processes. 

The term "data processing system" is used herein to refer to any machine for processing data, 
25 including the computer systems and network arrangements described herein. The term "Elastic 
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Presentation Space" ("EPS") (or "Pliable Display Technology" ("PDT")) is used herein to refer 
to techniques that allow for the adjustment of a visual presentation without interfering with die 
information content of the representation. The adjective "elastic" is included in the term as it 
implies the capability of stretching and deformation and subsequent return to an original shape. 
5 EPS graphics technology is described by Caipendale in "A Framework for Elastic Presentation 
Space" (Caipendale, Marianne S. T., A Framework for Elastic Presentation Space (Buxnaby, 
British Columbia: Simon Fraser University, 1999)), which is incorporated herein by reference. In 
EPS graphics technology, a two-dimensional visual representation is placed onto a surface; this 
surface is placed in three-dimensional space; the surface, containing the representation, is viewed 
10 through perspective projection; and the surface is manipulated to effect the reorganization of 
image details. The presentation transformation is separated into two steps: surface manipulation 
or distortion and perspective projection. 

FIG. 1 is a graphical representation 100 of the geometry for constructing a three-dimensional 
("3D") perspective viewing frustum 220, relative to an x, y, z coordinate system, in accordance 

15 with known elastic presentation space (EPS) graphics technology. In EPS technology, detail-in- 
context views of two-dimensional ("2D") visual representations are created with sight-line 
aligned distortions of a 2D information presentation surface within a 3D perspective viewing 
frustum 220. In EPS, magnification of regions of interest and the accompanying compression of 
the contextual region to accommodate this change in scale are produced by the movement of 

20 regions of the surface towards the viewpoint ("VP") 240 located at the apex of the pyramidal 
shape 220 containing the frustum. The process of projecting these transformed layouts via a 
perspective projection results in a new 2D layout which includes the zoomed and compressed 
regions. The use of the third dimension and perspective distortion to provide magnification in 
EPS provides a meaningful metaphor for die process of distorting the information presentation 

25 surface. The 3D manipulation of the information presentation surface in such a system is an 
intermediate step in the process of creating a new 2D layout of the information. 

FIG. 2 is a graphical representation 200 of die geometry of a presentation in accordance with 
known EPS graphics technology. EPS graphics technology employs viewer-aligned perspective 
projections to produce detail-in-context presentations in a reference view plane 201 which may 
30 be viewed on a display. Undistorted 2D data points are located in a basal plane 210 of a 3D 
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perspective viewing volume or frustum 220 which is defined by extreme rays 221 and 222 and 
the basal plane 210. The VP 240 is generally located above the centre point of the basal plane 
210 and reference view plane ("RVP") 201. Points in the basal plane 210 are displaced upward 
onto a distorted surface 230 which is defined by a general 3D distortion function (i.e. a detail-in- 
5 context distortion basis function). The direction of the viewer-aligned perspective projection 
corresponding to the distorted surface 230 is indicated by the line FPo - FP 231 drawn from a 
point FPo 232 in the basal plane 210 through the point FP 233 which corresponds to the focus or 
focal region or focal point of the distorted surface 230. 

EPS is applicable to multidimensional data and is well suited to implementation on a computer 

10 for dynamic detail-in-context display on an electronic display surface such as a monitor. In the 
case of two dimensional data, EPS is typically characterized by magnification of areas of an 
image where detail is desired 233, in combination with compression of a restricted range of areas 
of the remaining information (i.e. the context) 234, the end result typically giving the appearance 
of a lens 230 having been applied to the display surface. The areas of the lens 230 where 

IS compression occurs may be referred to as die "shoulder" 234 of the lens 230. The area of the 
representation transformed by the lens may be referred to as the "lensed area 11 . The lensed area 
thus includes the focal region and the shoulder. To reiterate, the source image or representation 
to be viewed is located in the basal plane 210. Magnification 233 and compression 234 are 
achieved through elevating elements of the source image relative to the basal plane 210, and then 

20 projecting the resultant distorted surface onto the reference view plane 201 . EPS performs detail- 
in-context presentation of n-dimensional data through the use of a procedure wherein the data is 
mapped into a region in an (n+1) dimensional space, manipulated through perspective 
projections in the (n+1) dimensional space, and then finally transformed back into n-dimensional 
space for presentation. EPS has numerous advantages over conventional zoom, pan, and scroll 

25 technologies, including the capability of preserving the visibility of information outside 234 the 
local region of interest 233. 

For example, and referring to FIGS. 1 and 2, in two dimensions, EPS can be implemented 
through the projection of an image onto a reference plane 201 in the following manner. The 
source image or representation is located on a basal plane 210, and those regions of interest 233 
30 of the image for which magnification is desired are elevated so as to move them closer to a 
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reference plane situated between (he reference viewpoint 240 and the reference view plane 201. 
Magnification of the focal region 233 closest to the RVP 201 varies inversely with distance from 
the RVP 201. As shown in FIGS. 1 and 2, compression of regions 234 outside the focal region 
233 is a function of both distance from the RVP 201, and the gradient of the function describing 
5 the vertical distance from the RVP 201 with respect to horizontal distance from the focal region 
233. The resultant combination of magnification 233 and compression 234 of the image as seen 
from the reference viewpoint 240 results in a lens-like effect similar to that of a magnifying glass 
applied to the image. Hence, the various functions used to vary the magnification and 
compression of the source image via vertical displacement from the basal plane 210 are 
10 described as lenses, lens types, or lens functions. Lens functions that describe basic lens types 
with point and circular focal regions, as well as certain more complex lenses and advanced 
capabilities such as folding, have previously been described by Carpendale. 

System. FIG. 3 is a block diagram of an exemplary data processing system 300 for implementing 
an embodiment of the invention. The data processing system is suitable for implementing EPS 

IS technology and for viewing detail-in-context presentations in conjunction with a graphical user 
interface ("GUT). The data processing system 300 includes an input device 310, a central 
processing unit or CPU 320, memory 330, and a display 340. The input device 310 may include 
a keyboard, mouse, trackball, or similar device. The CPU 320 may include dedicated 
coprocessors and memory devices. The memory 330 may include RAM, ROM, databases, or 

20 disk devices. And, the display 340 may include a computer screen or terminal device. The data 
processing system 300 has stored therein data representing sequences of instructions which when 
executed cause the method described herein to be performed. Of course, the data processing 
system 300 may contain additional software and hardware a description of which is not 
necessary for understanding the invention. 

25 GUI Using Detail-In-Context Lenses. According to an embodiment of the invention, the use of 
detail-in-context lenses for direct interaction with digital media and for interfacing with network 
portals is provided. 

According to an embodiment of the invention, detail-in-context lenses act as portals or interfaces 
to networked and local sources of information including shopping networks, financial networks, 
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and local information sources, in the following manner. When a lens is placed over an area of 
interest, a user can interact with the lens directly either via a pointing device (such as a mouse or 
stylus) or a remote control device (such as a TV remote control or some other wireless 
controlling device such as a tablet or handheld computer).Using a lens in this instance allows a 
5 lens to be used during a TV show to view items of interest in much greater detail without ever 
leaving the show. In addition to viewing the item in much greater detail, in the presort invention, 
the user can select and directly interact with the item of interest For example, a TV views 
watching a show can view a watch that a celebrity is wearing in real-time and in much greater 
detail, without ever leaving the show. This interaction is much different than simply clicking on 
10 die watch and having a window popup with a stale picture of the watch. With the lens, a user 
views the exact item as it appears on screen in real time. Furthermore, the lens may move in real- 
time with the item it is associated with. 

Using a remote controlling device, the user can directly control the lens and place it over 
different items of interest or have the lens scripted to move over different items. Multiple lenses 
15 can be used and controlled In addition, using the lens interface, not only can the item of interest 
be viewed in greater detail but, additional details about the item can be viewed (for example, the 
brand, model, and price). 

The present invention includes a mechanism to allow this watch to be purchased directly or 
placed in a "shopping cart" for later purchasing. The lens can also interface with stored user 
20 profiles. These profiles, which store user detials (eg. account and payment details), can be read 
by the lens to submit as identification to the shopping network. In addition to viewing details and 
interacting with details in digital media, the lens also serves as a highlighting medium to draw 
attention to items. 

The lens in the present invention is not limited to simply television shows but can be used in 
25 other media such as DVDs and video games. For instance, a lens can move with the action in a 
soccer video game such that paid sponsors can have their banners in die background be 
highlighted and given attention as the game progresses. 

The present invention may be used with Windows XP Media Center, for example, for: detail-in- 
context viewing of television programs; interaction with live television programs; and, 
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encouraging purchases within television programs without leaving the show; and, shopping 
within television shows and interfacing PDT with MSN Shopping via .NET Web Services. With 
the present invention, a user may: watch a favorite television show; view a watch worn by a 
favorite celebrity with great detail using PDT, within the context of television program; and, gain 
5 instant access to purchases via MSN and .NET Web Services with PDT 1ms interface and stored 
Passport profiles. For example, a user may use a remote control or tablet control to active PDT, 
clearly view an item of interest, and purchase the item via MSN Shopping, all without leaving 
the context of the show or its celebrities. 

The present invention may also be used with gaming consoles such as Microsoft's Xbox. These 
10 gaming consoles are often positioned as general-purpose entertainment devices and may benefit 
from the application of PDT. For example, with respect to Xbox moving towards a more general- 
purpose home entertainment unit, PDT provides the following functionality gains: enables more 
powerful online collaboration and multiplayer gaming capabilities; digital image editing and 
management; interactive television/web browsing interactive advertising; and, can work as a 
IS narrative/presentation tool or scripted communication tool. 

With respect to Xbox information storage, cataloging, and data management, PDT provides the 
following functionality gains: digital image management; viewing of different content layers 
and overlays brought into the lens at various scales; and, more efficient navigation of "cluttered" 
information, cataloging of song titles, movie titles, etc. 

20 With respect to Xbox image manipulation and presentation tools, PDT provides the following 
functionality gains: a powerful presentation tool that highlights area of interest; editing of images 
and models; and, 2D and 3D narration or scripting. 

With respect to Xbox Product's Developer, Testing, Animation, Designs* and Authoring Tools, 
appli d PDT functionality provides: efficient detail-in-context viewing and in-place editing 
25 workspace for people that are spatially restricted within their on-screen 2D and 3D workspaces; 
and, a presentation/communication tool to help create, modify, test, and inspect graphics for 
quality control purposes. 
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With respect to Xbox Product's video games to enhance gaming experience (i.e. Strategy Puzzle, 
First Person Shooter, Role Playing, Flight Simulator, Action/Adventure, etc.), applied PDT 
functionality provides: use of PDT for "cool" visual effects; increasing functionality of editing 
packages and tools used to create new worlds; adding to existing zooming and panning 
5 capabilities within games; and, an exciting new mode of co-located multiplayer and online 
multiplayer interactivity. 

FIG. 6 is a screen capture illustrating the application of lenses in multi-player video games in 
accordance with an embodiment of the invention. With respect to Xbox Live and PDT for multi- 
player collaiboration, PDT provides: an exciting new mode of co-located multiplayer and online 

10 multiplayer interactivity; cool new special effects that are ideal for enhancement of narrative 
capabilities, increasing the draw-in rate of first person shooter games to see further in front of 
view, and detail-in-context viewing for role playing games where players are required to be 
aware of their surroundings at all times; increased functionality of editing packages and tools 
available to create and explore new worlds; additions to existing zooming and panning 

15 capabilities; and, as games migrate to handsets and PDA's, PDT can aid in the change of content 
and display of games. 

With respect to Xbox and advertising, for example, a scripted lens can move with the action to 
highlight advertising banners for paid sponsors. FIG. 5 is a screen capture illustrating a lens 
applied to advertisements (i.e. a "Yahoo" banner) in a video game in accordance with an 
20 embodiment of die invention. As the game player progresses towards the goal, sequential 
banners (i.e. the "NBA" and "Toshiba" banners) are magnified or hi-lited by the lens. In this 
way; die advertisements follow the game play. Alternatively, a selected banner may be stretched 
or deformed in the direction of play. 

Data Carrier Product The sequences of instructions which when executed cause the method 
25 described herein to be performed by the exemplary data processing system of FIG. 3 can be 
contained in a data carrier product according to one embodiment of the invention. This data 
carrier product can be loaded into and run by die exemplary data processing system of FIG. 3 . 

Computer Software Product. The sequences of instructions which when executed cause Ae 
method described herein to be performed by die exemplary data processing system of FIG, 3 can 
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be contained in a computer software product according to one embodiment of the invention. This 
computer software product can be loaded into and run by the exemplary data processing system 
of FIG. 3. 

Integrated Circuit Product. The sequences of instructions which when executed cause the 
5 method described herein to be performed by the exemplary data processing system of FIG. 3 can 
be contained in an integrated circuit product including a coprocessor or memory according to one 
embodiment of the invention. This integrated circuit product can be installed in the exemplary 
data processing system of FIG. 3. 
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